Discriminating synonymous expressions using images

ABSTRACT

A method for identifying synonymous expressions includes determining synonymous expression candidates for a target expression. A plurality of target images related to the target expression and a plurality of candidate images related to each of the synonymous expression candidates are identified. Features extracted from the plurality of target images are compared with features extracted from the plurality of candidate images using a processor to identify a synonymous expression of the target expression.

BACKGROUND

1. Technical Field

The present invention relates to natural language processing, and moreparticularly to identifying synonymous expressions of words byconsidering the similarities between images.

2. Description of the Related Art

The identification of synonymous expressions is one of the mostimportant issues in natural language processing for handling textualdata. For example, the expression “windshield” can also be expressed as“windscreen” in English, or as “parabrisas,” “windschutzscheibe,” etc.in other languages. The use of online dictionaries for identifyingsynonymous expressions is very limited as many expressions are used in avariety of different textual data contexts.

For various applications of natural language processing, including textmining, machine translation, and information retrieval, theidentification of synonymous expressions is often important in improvingthe performance of applications. Because of this, a variety oftechniques have been developed for extracting synonymous expressions.However, none of these current techniques are mature enough, and theiraccuracies are evaluated by checking whether the correct answer iscontained within a list of top N candidate expressions rather thanwhether the answer is correct or not.

One of the limitations of the current techniques is that they only relyon textual information. For example, from a textual data set in theautomotive domain, the expressions “door” and “mirror” may be extractedas synonymous expressions of “window” using the current approaches, asthey share similar contexts such as “open” and “break.”

SUMMARY

A method for identifying synonymous expressions includes determiningsynonymous expression candidates for a target expression. A plurality oftarget images related to the target expression and a plurality ofcandidate images related to each of the synonymous expression candidatesare identified. Features extracted from the plurality of target imagesare compared with features extracted from the plurality of candidateimages using a processor to identify a synonymous expression of thetarget expression.

A system for identifying synonymous expressions includes a candidateidentification module configured to determine synonymous expressioncandidates for a target expression. An image selection module isconfigured to identify a plurality of target images related to thetarget expression and a plurality of candidate images related to each ofthe synonymous expression candidates. A comparison module is configuredto compare features extracted from the plurality of target images withfeatures extracted from the plurality of candidate images using aprocessor to identify a synonymous expression of the target expression.

These and other features and advantages will become apparent from thefollowing detailed description of illustrative embodiments thereof,which is to be read in connection with the accompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

The disclosure will provide details in the following description ofpreferred embodiments with reference to the following figures wherein:

FIG. 1 is a high-level block/flow diagram showing a system/method foridentifying synonymous expressions from a target expression, inaccordance with one illustrative embodiment;

FIG. 2 is a block/flow diagram showing a system for identifyingsynonymous expressions, in accordance with one illustrative embodiments;and

FIG. 3 is a block/flow diagram showing a method for identifyingsynonymous expressions, in accordance with one illustrative embodiment.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

In accordance with the present principles, systems and methods fordiscriminating synonymous expressions using images are provided. Thelinkage of images and textual information has been developed to apractical level in recent years. The present invention identifiessynonymous expressions of words by considering the similarities betweenthe images that are related to the words.

For a given target expression, synonymous expression candidates areidentified. A plurality of target images related to the targetexpression and a plurality of candidate images related to the synonymousexpression candidates are selected. Features are extracted for theimages, which may include using a plurality of feature extractiontechniques. The features of the target images are compared with thefeatures of the candidate images to identify a synonymous expression ofthe target expression. The present invention was found to have greateraccuracy compared to current techniques.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readablestorage medium. A computer readable storage medium may be, for example,but not limited to, an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system, apparatus, or device, or any suitablecombination of the foregoing. More specific examples (a non-exhaustivelist) of the computer readable storage medium would include thefollowing: an electrical connection having one or more wires, a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), an optical fiber, a portable compact disc read-onlymemory (CD-ROM), an optical storage device, a magnetic storage device,or any suitable combination of the foregoing. In the context of thisdocument, a computer readable storage medium may be any tangible mediumthat can contain, or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing. Computer program code for carrying out operations foraspects of the present invention may be written in any combination ofone or more programming languages, including an object orientedprogramming language such as Java, Smalltalk, C++ or the like andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The program codemay execute entirely on the user's computer, partly on the user'scomputer, as a stand-alone software package, partly on the user'scomputer and partly on a remote computer or entirely on the remotecomputer or server. In the latter scenario, the remote computer may beconnected to the user's computer through any type of network, includinga local area network (LAN) or a wide area network (WAN), or theconnection may be made to an external computer (for example, through theInternet using an Internet Service Provider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks. The computer program instructions may also beloaded onto a computer, other programmable data processing apparatus, orother devices to cause a series of operational steps to be performed onthe computer, other programmable apparatus or other devices to produce acomputer implemented process such that the instructions which execute onthe computer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblocks may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

Reference in the specification to “one embodiment” or “an embodiment” ofthe present principles, as well as other variations thereof, means thata particular feature, structure, characteristic, and so forth describedin connection with the embodiment is included in at least one embodimentof the present principles. Thus, the appearances of the phrase “in oneembodiment” or “in an embodiment”, as well any other variations,appearing in various places throughout the specification are notnecessarily all referring to the same embodiment.

It is to be appreciated that the use of any of the following “/”,“and/or”, and least one of”, for example, in the cases of “A/B”, “Aand/or B” and “at least one of A B”, is intended to encompass theselection of the first listed option (A) only, or the selection of thesecond listed option (B) only, or the selection of both options (A andB). As a further example, in the cases of “A, B, and/or C” and “at leastone of A, B, and C”, such phrasing is intended to encompass theselection of the first listed option (A) only, or the selection of thesecond listed option (B) only, or the selection of the third listedoption (C) only, or the selection of the first and the second listedoptions (A and B) only, or the selection of the first and third listedoptions (A and C) only, or the selection of the second and third listedoptions (B and C) only, or the selection of all three options (A and Band C). This may be extended, as readily apparent by one of ordinaryskill in this and related arts, for as many items listed.

Referring now to the drawings in which like numerals represent the sameor similar elements and initially to FIG. 1, a high-level block/flowdiagram showing a system/method for identifying synonymous expressions100 is shown in accordance with one illustrative embodiment. For atarget word 102, denoted as W_(o), candidates words 104 of synonymousexpressions {S_(j): j=1, . . . , m} are selected. For the target wordW_(o) and each candidate S_(j), multiple related images P_(i) andQ_(jk), respectively, are selected. Features are extracted for eachimage and the features for target word images P_(i) are compared withfeatures for candidate word images Q_(jk) to identify the image whosefeatures are most similar to extract a synonymous expression for thetarget word W_(o).

Referring now to FIG. 2, a block/flow diagram showing a system foridentifying synonymous expressions 200 is shown in accordance with oneillustrative embodiment. The system 200 identifies synonymousexpressions of words by considering the similarities between the imagesthat are related to the words.

It should be understood that embodiments of the present principles maybe applied in a number of different applications. For example, thepresent invention may be discussed throughout this application as interms of identifying synonymous expressions. However, it should beunderstood that the present invention is not so limited. Rather,embodiments of the present principles may be employed in any applicationto identify similar or related candidates for any target.

The system 200 may include a system or workstation 202. The system 202preferably includes one or more processors 208 and memory 210 forstoring data, applications, modules and other data. The system 202 mayalso include one or more displays 204 for viewing. The displays 204 maypermit a user to interact with the system 202 and its components andfunctions. This may be further facilitated by a user interface 206,which may include a mouse, joystick, or any other peripheral or controlto permit user interaction with the system 202 and/or its devices. Itshould be understood that the components and functions of the system 202may be integrated into one or more systems or workstations, or may bepart of a larger system or workstation.

The system 202 may receive input 212, which may include targetexpression 214. The target expression 214 may include one or more words,phrases, images, sounds, or other forms of expressions that synonymousexpressions are to be identified for. The synonymous expressions mayinclude one or more synonymous or similar words, phrases, images,sounds, etc., which may be in another language, format, orconfiguration.

The candidate identification module 216 is configured to identifycandidates of synonymous expressions for the target expression 214. Thecandidates of synonymous expressions may be identified by knowntechniques. For example, see commonly assigned U.S. Pat. No. 7,483,829entitled “Candidate synonym support device for generating candidatesynonyms that can handle abbreviations, mispellings, and the like;” U.S.Pat. No. 8,463,794 entitled “Computer system, method, and computerprogram for extracting terms from document data including text segment;”and U.S. patent application Ser. No. 13/413,866 entitled “Method,program and system for finding correspondence between terms,” all ofwhich are incorporated herein by reference in their entirety. Othertechniques may also be employed within the context of the presentinvention.

The image selection module 218 is configured to select a plurality ofimages that are related to the target expression and each candidate.Known search engines may be employed to select the plurality of images.For example, image meta search or content-based image retrieval (CBIR)techniques may be employed to select a plurality of images related tothe target expression and candidate synonym expressions. Othertechniques are also contemplated.

The comparison module 220 is configured to compare features extractedfrom the target expression images with the candidate expression imagesto determine synonymous expressions of the target expression. Featuresmay be extracted from each of the selected plurality of images usingknown feature extraction techniques, such as, e.g., speeded up robustfeatures (SURF), scale-invariant feature transform (SIFT), orientedBRIEF (ORB), etc. Other feature extractions techniques may also beemployed. Preferably, multiple feature extraction techniques areemployed to extract features from the plurality of images.

In some embodiments, color information may be useful as an imagefeature. For example, color information may be relevant where the targetword is “fire.” To incorporate color information as an image feature,the binary data (e.g., 10 bit) of the RGB color model is determined andexpressed as a histogram of 1000 dimensions (i.e., 10×10×10).

Similarities are calculated for the target expression images. Outliersthat are not similar to the other images are removed using outlierdetection techniques. Outlier detection techniques may include, e.g.,local outlier factor (LOF), student's t-test, etc. Other techniques mayalso be employed. A method for extracting features and calculatingsimilarity is selected that is best fitted and results in the highestsimilarities among the remaining target expression images.

Using the selected feature extraction and similarity method, pair-wisesimilarities s(l,j,k) are calculated between each image from theremaining target expression images and each candidate expression image.The similarities preferably include, e.g., cosine similarity, Jaccardsimilarity, etc. Calculating similarity may include extracting visualwords by clustering all the keypoints from feature extraction of thecandidate expression images, attach cluster IDs (i.e., visual words) foreach keypoint in target expression images and candidate expressionimages, calculating a histogram of visual words for each image as afeature vector, and calculating cosine similarity of feature vectors foreach pair of images (target expression image and candidate expressionimage).

In some embodiments, a distance measure may be better than thesimilarity measure. For example, a distance measure may be a bettermeasure where the target expression is “injury.” The distance measuremay include, e.g., a Euclidean distance, Manhattan distance, etc. Otherdistance metrics may also be employed. The distance of each image issorted in, e.g., ascending order, where each distance is calculated fromthe histogram of feature extraction.

The pair-wise similarities s(l,j,k) are ranked, e.g., in descendingorder and the number of times each candidate synonymous expressionappears above a threshold (i.e., the frequency) is determined. Thecandidate synonymous expression appearing most frequently above thethreshold is determined as the synonymous expression 226 of the targetimage.

The synonymous expression 226 may be included as the output 224. Thesynonymous expression 226 may include one or more synonymous or similarwords, phrases, images, sounds, etc., which may be in another language,format, or configuration.

Referring now to FIG. 3, a block/flow diagram showing a method forsynonymous expression identification 300 is depicted in accordance withone illustrative embodiment. In block 302, synonym expression candidatesare determined for a target expression. The target expression manyinclude expressions (e.g., words, phrases, images, sounds, etc.) forwhich a synonymous expression is to be determined. In block 304, aplurality of target images related to the target expression and aplurality of candidate images relates to each of the synonym expressioncandidates are identified. Identifying the images may include employingimage meta search, CBIR, etc.

In block 306, features extracted from the plurality of target images arecompared with features extracted from the plurality of candidate imagesto identify a synonymous expression of the target expression. This mayinclude extracting features from the plurality of target images andplurality of candidate images in block 308. Feature extraction mayemploy, e.g., SURF, SIFT, ORB, etc. In some embodiment, extractingfeatures may include extracting color information as an image feature.

In block 310, a metric is employed to calculate a similarity betweeneach of the plurality of target images and each of the plurality ofcandidate images. Preferably, the metric includes a similarity metric,such as, e.g., cosine similarity, Jaccard similarity, etc. In oneembodiment, the metric may also include a distance metric, such as,e.g., a Euclidean distance, Manhattan distance, etc. Other metrics mayalso be employed within the context of the present principles.

In some embodiments, outliers of the plurality of target images areeliminated before the similarity between each of the plurality of targetimages and each of the plurality of candidate images is calculated. Thismay include calculating similarities between target images and removingoutliers using outlier detection techniques. In a preferred embodiment,a plurality of feature extraction methods and a plurality of metrics areemployed and the feature extraction method and metric (e.g., similarity,distance) is selected that is best fitted and results in the highestsimilarity between the target expression images.

In block 312, the synonymous expression of the target expression isidentified based on the similarity. The synonymous expression mayinclude expressions (e.g., words, phrases, images, sounds, etc.) whichare synonymous with the target expression. In one embodiment,identifying the synonymous expression may include sorting the synonymousexpression candidates associated with the candidate images andidentifying the most frequent synonymous expression candidate as thesynonymous expression. A threshold may be employed to identify the mostfrequent synonymous expression candidates above a threshold as thesynonymous expression. In another embodiment, identifying the synonymousexpression may include identifying the top N synonymous expressioncandidates associated with candidate images having the top N highestsimilarity or distance. Other methods of identifying the synonymousexpression based on the metric are also contemplated.

Having described preferred embodiments of a system and method fordiscriminating synonymous expressions using images (which are intendedto be illustrative and not limiting), it is noted that modifications andvariations can be made by persons skilled in the art in light of theabove teachings. It is therefore to be understood that changes may bemade in the particular embodiments disclosed which are within the scopeof the invention as outlined by the appended claims. Having thusdescribed aspects of the invention, with the details and particularityrequired by the patent laws, what is claimed and desired protected byLetters Patent is set forth in the appended claims.

What is claimed is:
 1. A method for identifying synonymous expressions,comprising: determining synonymous expression candidates for a targetexpression; identifying a plurality of target images related to thetarget expression and a plurality of candidate images related to each ofthe synonymous expression candidates; and comparing features extractedfrom the plurality of target images with features extracted from theplurality of candidate images using a processor to identify a synonymousexpression of the target expression.
 2. The method as recited in claim1, wherein comparing includes employing a metric to compare featuresextracted from the plurality of target images with features extractedfrom the plurality of candidate images.
 3. The method as recited inclaim 2, wherein the metric includes at least one of a similarity metricand a distance metric.
 4. The method as recited in claim 2, whereincomparing further comprises identifying the synonymous expression of thetarget expression based on the metric.
 5. The method as recited in claim4, wherein comparing further comprises ranking the images based on themetric and selecting a synonymous expression candidate associated with amost frequent candidate image as the synonymous expression.
 6. Themethod as recited in claim 5, wherein selecting includes selecting thesynonymous expression candidate associated with the most frequentcandidate image above a threshold as the synonymous expression
 7. Themethod as recited in claim 1, wherein the features extracted from theplurality of target images and the features extracted from the pluralityof candidate images includes color information.
 8. The method as recitedin claim 1, further comprising removing outliers from the plurality oftarget images.
 9. The method as recited in claim 1, wherein thesynonymous expression includes at least one of a word, phrase, image,and sound.
 10. A non-transitory computer readable storage mediumcomprising a computer readable program for identifying synonymousexpressions, wherein the computer readable program when executed on acomputer causes the computer to perform the steps of: determiningsynonymous expression candidates for a target expression; identifying aplurality of target images related to the target expression and aplurality of candidate images related to each of the synonymousexpression candidates; and comparing features extracted from theplurality of target images with features extracted from the plurality ofcandidate images to identify a synonymous expression of the targetexpression.
 11. The non-transitory computer readable storage medium asrecited in claim 10, wherein comparing includes employing a metric tocompare features extracted from the plurality of target images withfeatures extracted from the plurality of candidate images, the metricincluding at least one of a similarity metric and a distance metric. 12.The non-transitory computer readable storage medium as recited in claim10, wherein the features extracted from the plurality of target imagesand the features extracted from the plurality of candidate imagesincludes color information.
 13. A system for identifying synonymousexpressions, comprising: a candidate identification module configured todetermine synonymous expression candidates for a target expression; animage selection module configured to identify a plurality of targetimages related to the target expression and a plurality of candidateimages related to each of the synonymous expression candidates; and acomparison module configured to compare features extracted from theplurality of target images with features extracted from the plurality ofcandidate images using a processor to identify a synonymous expressionof the target expression.
 14. The system as recited in claim 13, whereinthe comparison module is further configured to employ a metric tocompare features extracted from the plurality of target images withfeatures extracted from the plurality of candidate images.
 15. Thesystem as recited in claim 14, wherein the metric includes at least oneof a similarity metric and a distance metric.
 16. The system as recitedin claim 14, wherein the comparison module is further configured toidentify the synonymous expression of the target expression based on themetric.
 17. The system as recited in claim 16, wherein the comparisonmodule is further configured to rank the images based on the metric andselect a synonymous expression candidate associated with a most frequentcandidate image as the synonymous expression.
 18. The system as recitedin claim 17, wherein the comparison module is further configured toselect the synonymous expression candidate associated with the mostfrequent candidate image above a threshold as the synonymous expression19. The system as recited in claim 13, wherein the features extractedfrom the plurality of target images and the features extracted from theplurality of candidate images includes color information.
 20. The systemas recited in claim 13, wherein the comparison module is furtherconfigured to remove outliers from the plurality of target images.